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ABSTRACT 

NASA is researching innovative technologies for the Next Generation Air Transportation System (NextGen) to 
provide a “Better-Than-Visual” (BTV) capability as adjunct to “Equivalent Visual Operations” (EVO); that 
is, airport throughputs equivalent to that normally achieved during Visual Flight Rules (VFR) operations rates 
with equivalent and better safety in all weather and visibility conditions including Instrument Meteorological 
Conditions (IMC). These new technologies build on proven flight deck systems and leverage synthetic and 
enhanced vision systems. Two piloted simulation studies were conducted to access the use of a Head- Worn Display 
(HWD) with head tracking for synthetic and enhanced vision systems concepts. The first experiment evaluated 
the use a HWD for equivalent visual operations to San Francisco International Airport (airport identifier: KSFO) 
compared to a visual concept and a head-down display concept. A second experiment evaluated symbology 
variations under different visibility conditions using a HWD during taxi operations at Chicago O’Hare airport 
(airport identifier: KORD). 

Two experiments were conducted, one in a simulated San Francisco airport (KSFO) approach operation and 
the other, in simulated Chicago O’Hare surface operations, evaluating enhanced/synthetic vision and head-worn 
display technologies for NextGen operations. While flying a closely-spaced parallel approach to KSFO, pilots 
rated the HWD, under low-visibility conditions, equivalent to the out-the-window condition, under unlimited 
visibility, in terms of situational awareness (SA) and mental workload compared to a head-down enhanced vision 
system. There were no differences between the 3 display concepts in terms of traffic spacing and distance and 
the pilot decision-making to land or go-around. For the KORD experiment, the visibility condition was not a 
factor in pilot’s rating of clutter effects from symbology. Several concepts for enhanced implementations of an 
unlimited field-of-regard BTV concept for low-visibility surface operations were determined to be equivalent in 
pilot ratings of efficacy and usability. 
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1. INTRODUCTION 

Adverse weather accounts for in approximately 70% of all flight delays in the National Airspace System (NAS). 
Air traffic operations in the terminal maneuvering area are particularly affected by Instrument Meteorological 
Conditions (IMC). Estimates posit that airport throughput may drop as much as a factor of 3 during periods 
of low visibility that can compound into delays throughout the NAS. Existing technologies and new airport 
construction marginally reduce the “performance gap” for throughput between IMC and Visual Meteorological 
Conditions (VMC) at the 35 major US commercial hub airports. For example, a new runway at Atlanta (ATL) 
increased airport throughput by only 3%J In any event, for some airports, construction of new runways is not 
possible and instead must rely solely on technology innovation to reduce the performance gap. 
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The Vehicle Systems and Safety Technologies (VSST) project, under NASAs Aviation Safety Program, is 
conducting research, development, test and evaluation of crew-vehicle interface technologies to ensure effective 
crew-system interactions and decisions in all conditions (operational and environmental). One goal is to proac- 
tively overcome aircraft safety barriers that might otherwise constrain the full realization of the Next Generation 
Air Transportation System (NextGen). Part of this research effort involves the use of synthetic and enhanced 
vision systems and advanced display media as enabling crew-vehicle interface technologies to meet these safety 
challenges. 

One emerging NextGen concept termed, “Equivalent Visual Operations” (EVO), can be achieved using an 
electronic means to provide sufficient visibility of the external world and other required flight references on flight 
deck displays that enable the safety, operational tempos, and VMC-like procedures for all weather conditions. 
Synthetic Vision (SV) and Enhanced Vision System (EVS) technologies are critical enabling technologies to 
EVO with the ultimate objective of EVO, being one component of a “Better-Than-Visual” (BTV) operational 
capability; replicating the capacity of today’s VMC flight and more importantly, meeting and improving on the 
safety of today’s VMC flight in all-weather NextGen operations. 

Synthetic and Enhanced Vision (S/EV) technologies are emerging as standard equipage on today’s flight 
deck. 2 S/EV generates intuitive visual references for the flight crew/pilot to fly the aircraft as if in VMC 
independent of the actual visibility or lighting conditions. NASA research aims to extend the present-day 
S/EV concepts to enable VMC-like operational tempos and maintain and improve the safety of VMC while 
using VMC-like procedures in all-weather conditions. To meet this potential, research is focused on S/EV 
technology development and human-in-the-loop performance to enable a “visual” approach, landing, roll-out, 
and surface operations down to 300 ft actual Runway Visual Range (RVR). This BTV operational concept 
suggests that the minimum aviation system performance standard for BTV technologies should be that as 
defined by human performance in the same operation using windows during today’s VMC operations. Significant 
research is required to quantify this hypothesized performance standard, and more importantly, to determine 
if it is indeed an appropriate and sufficient standard for BTV. S/EV work includes the development of fusion 
methods for synthetic and enhanced vision systems; feature extraction by use of real-time imaging sensors; on- 
board navigational, sensor, and database integrity monitoring; and appropriate display methods for Head-Up 
Displays (HUDs) and Head- Worn Displays (HWDs), primary flight and navigation displays, and electronic flight 
bags. 

1.1 Head- worn display for commercial operations 

NASA has conducted numerous studies evaluating the potential benefits of using HWDs for surface operations. 3-5 
HWDs are small, light weight display devices that can be worn on the head without significant encumbrance 
(Fig. 1). By coupling the HWD with a head tracker, unlimited field-of-regard can be realized. For a SV-generated 
scene, the SV-viewpoint position and orientation can be defined via software; thus, an unlimited field-of-regard is 
achieved since the Synthetic Vision scene is viewable from any virtual camera angle. For sensor imagery, multiple 
tiled sensors or a “turreted” sensor would be required to realize a large field-of-regard. These implementations 
have found merits in military avionic applications. 6 

1.2 Airborne operations 

The BTV operational concept expands on procedural and technological innovations that are being developed 
to support NextGen operations, such as Flight deck Interval Management (FIM). FIM leverages advancements 
in communications, surveillance, and navigation to enable flight crews to precisely space their aircraft relative 
to another aircraft in the terminal maneuvering area. The goal is to improve airport throughput and reduce 
delays. Using FIM, the air traffic controller instructs the participating aircraft to achieve an assigned inter- 
arrival spacing interval at the runway threshold, relative to another aircraft, using on-board automation. The 
flight crew then takes responsibility to actively fly the FIM operation but the Air Navigation Services Provider 
(ANSP) retains the responsibility for aircraft separation. NASA research has demonstrated the efficacy of the 
concept and system- wide and algorithm effects. 7 

Research has recently been completed showing the synergistic potential of combining FIM and S/EV tech- 
nologies, broaching the concept of BTV. 8 Simulation testing showed the ability of flight crews to self-separate, 




Figure 1. The HWD system integrated with an aviation headset. 


wherein the pilot /flight crew accepted responsibility for separation from the designated paired aircraft, and 
maintained an “equivalent visual contact” through the use of ADS-B and S/EV technologies. Spacing intervals 
during self-separation approaches followed VMC-like operational profiles while maintaining a very high degree 
of flight precision, stabilized approach procedures, and excellent traffic/situation awareness. Further, ego- and 
exo-centric display concepts for terrain, traffic, and airport surface conditions kept the flight crew proactive in 
the operation and allowed them to easily manage the arrival, through landing (in simulated 700 RVR visibility 
conditions), roll-out, and turn-off with acceptable workload and sufficient spare attention/workload capacity to 
easily react to non-normal events that were intentionally staged in the experiment. 

Based on the success and efficacy of BTV for in-trail delegated separation and operational credit for lowered 
visibility landing minimums, the research has now turned to other operations that represent significant challenges 
for emerging NextGen operational concepts. A significant hurdle to overcoming capacity constraints are simul- 
taneous dependent parallel runway operations at runways separated by less than 4300 feet. In particular, very 
closely spaced parallel runways have equipment and procedural requirements which significantly limit their use 
in degraded visual conditions. For example, at San Francisco International Airport (airport identifier: KSFO), 
Runways 28L and 28R are separated by only 750 feet and the Simultaneous Offset Instrument Approach (SOIA) 
procedure has been created to maintain airport throughputs. However, at least 4 miles visibility and a 1200 
foot ceiling is required - just slightly less visibility conditions than Visual Flight Rules (VFR) operation. When 
the weather drops below these minima, the airport is reduced to single runway operations substantially reducing 
arrival rates from 65 to 30 per hour. 

KSFO operations are not atypical. To meet the “weather-independence” goal of NextGen, game-changing 
technology is needed to safely overcome these constraints. One goal of the research described herein is to conduct 
an initial evaluation of the use of the BTV concept, in concert with the FIM technologies, for very closely spaced 
simultaneous dependent parallel runway operations in IMC. To achieve the goal, a preliminary analysis was 
undertaken to assess the use and comparison of the KSFO SOIA under both simulated visual (better than 
current visibility minimums) and instrument conditions (simulated CAT I) using (a) “visual” out-the-window 
only or “equivalent visual” with either a (b) head-down or on a (c) head-worn display with synthetic and enhanced 
vision systems. 

1.3 Surface operations 

NASA has performed previous surface operations research using head-down displays, HUDs and HWDs. 5 This 
research has explored numerous operating paradigms and technologies, including the benefits of various display 
types versus paper charts, binocular versus monocular HWDs, 9 and flight data presentation (symbology) on the 



display. One goal of NASA’s research is to evaluate the efficacy and utility of HWD concepts to support NextGen 
surface operations. 

This study described herein is an extension of the head-up symbology research by exploring two augmented 
reality concepts which significantly modulate the HWD clutter or occlusion characteristics. HWDs have been 
shown to uniquely provide the display capabilities desired for piloted surface operations; that is, unlimited field- 
of-regard, head-up operations. The design challenge (and certification challenge) is to create this capability 
without pilot workload, encumbrance, or obscuration of their normal vision. 

From previous experiments, pilot comments have highlighted several possible methods to reduce the “clutter” 
of the overall HWD symbology. First, pilots suggested that the HWD symbology provide more augmented reality 
rather than virtual reality presentation. Pilots commented that an augmented reality concept is preferred over a 
virtual reality concept because if an object is visible in the real world, it does not need to be shown virtually on 
the HWD. Further, pilots have stressed that symbology content in the HWD should only be critical information. 
The rationale is to de-clutter the HWD by not obscuring visible objects (‘if I can see it, I don’t need to show a 
computer-generated representation on top of the visible object’). 

Figure 2 shows an augmented reality concept developed by NASA to reduce HWD clutter for surface opera- 
tions. The boxed area containing symbology and the virtual airport represents the view as it would be rendered 
on the HWD. Outside the boxed area represents the visibility of the actual airport environment with natural 
vision. In this example, the figure represents a reported visibility of 700 RVR. The augmented reality concept 
draws the S/EV imagery (i.e. the ‘virtual airport environment’) beyond the reported 700 RVR, a so called 
“Beyond-RVR” representation. Note that the SV airport is not shown in the HWD up to 700 RVR as this por- 
tion of the actual airport can be seen with the unaided eye. Essentially, the SV airport was culled up to 700 RVR 
(approximately the lower sixth of the display in Fig. 2) to allow pilots to view the actual airport environment. 
The cleared route (shown as the magenta ribbon) was drawn on top of the yellow taxiway centerline to denote 
the cleared path. The cleared path always overlays the taxiway centerline if in view on the HWD as it is critical 
information for runway and taxiway incursion prevention. 10, 11 



Figure 2. The HWD symbology concept for surface operations. 

A second augmented reality symbology technique was also examined. In previous experiments, 5 generic 
aircraft models were used to denote traffic on the scene rendered in the HWD. The concept used a variable-size, 
based on range, unfilled diamond shape rendered at the reported traffic location. This unfilled diamond shape 
has the advantage of a low clutter “footprint” but lacks traffic heading information or other information possibly 
derived by using an aircraft model depiction. 





This study examined these 2 augmented reality techniques (Beyond-RVR and traffic rendering) under 3 
visibility conditions. The first augmented technique modulated the amount of SV airport environment that 
was rendered in the HWD. This was compared to a S/EV-generated airport environment that was rendered 
across the entire HWD field-of-view (used in previous experiments). The second augmented reality technique 
modulated the way other aircraft were rendered in the HWD. Traffic was rendered as symbology on the HWD 
as either a generic model aircraft or an unfilled diamond. 

2. EXPERIMENTS 

Two experiments were conducted to determine the efficacy of using a head tracked HWD to explore equivalent 
visual operations for a closely spaced parallel approach and to explore augmented reality techniques for sur- 
face operations. The experiments were conducted back-to-back using the same HWD, simulation facility and 
Evaluation Pilots (EPs). 

2.1 Head- worn display 

The HWD used in both experiments was a prototype Rockwell Collins ProView™ S035-MTV 12 (Fig. 3). This 
display was a transparent 800x600 pixel, monochrome green display. The field-of-view (FOV) of the display 
was measured to be 28.6° H by 21.7° V. Coupled to this display was a prototype head tracker by Intersense, 
Inc., based on the IS-1200 tracker. The head tracker system used a hybrid combination of optical and inertial 
sensors. Both the display and tracker were mounted to clear lens glasses. The clear lens glasses were large 
enough to accommodate most eye wear. The head-borne weight of the combined display and tracker system was 
approximately 180 grams. 



Figure 3. The HWD system. 


2.2 Simulation facility 

Both studies were conducted in the Visual Imaging Simulator for Transport Aircraft Systems (VISTAS) III 
simulator (Fig. 4) at NASA Langley Research Center (LaRC). VISTAS III is a single pilot fixed-based simulator 
consisting of a 144 degree by 30 degree out the window visual, large field head-down display (HDD) and pilot 
input controls. The simulated aircraft in VISTAS III was a large commercial transport type aircraft. 




Figure 4. VISTAS III simulator. 


2.3 Evaluation pilots 

Nine commercial EPs from 5 different US airlines participated in both experiments. Five EPs were current or 
recently retired captains and 4 EPs were current first officers. 

The EPs were given a 45-minute briefing to explain the display concepts and the evaluation tasks for both 
experiments. After the briefing, a 30-minute training session was conducted to familiarize the EPs with the 
airborne portion of the experiment. EPs were also shown a 10 minute video describing the current KSFO 
SOIA procedure. Following this training, 1.5 hours of data collection was conducted. After the airborne data 
collection, a 1-hour training session was conducted to familiarize the EPs with the surface operations portion of 
the experiment. Following the surface training, data collection trials lasted 1.5 hours. The total time for an EP 
was approximately 6 hours. 

3. EXPERIMENT ONE: AIRBORNE OPERATIONS 

3.1 Methodology 

The potential application of BTV technology was explored using simulated KSFO operations. The experiment 
evaluated the potential of vision-based technology to overcome safety barriers in dependent closely spaced parallel 
runway operations. To determine initial feasibility, this research evaluated the capability of BTV technologies to 
enable “equivalent visual” parallel runway operations in IMC. The SOIA at KSFO was chosen as the scenario 
test case. 

3.1.1 Evaluation task 

Each EP flew as the monitoring pilot while conducting the Localizer Type Directional Aid (LDA) Precision 
Runway Monitor (PRM) 28R approach to KSFO. Parallel traffic flew the ILS PRM 28L approach. The own-ship 
aircraft (large commercial transport) was flown using the autopilot coupled to the approach to Runway 28R. The 
operation was also paired using FIM with traffic approximately 2 minutes ahead. Automated audio call-outs 
typically made by a first officer were provided to provide awareness to the aircraft configuration. Other simulated 
audio events included Air Traffic Control (ATC) instructions, traffic party line, and aircraft noises (engines, flap 
servos, landing gear deployment). 

The SOIA requires visual contact with both the airport environment and parallel traffic on 28L by the 
Decision Altitude (DA). Prior to the DA, ATC requested own-ship to respond as to whether the parallel traffic 
was in sight. EPs were instructed to answer ATC with “equivalent visual contact” if they could see the parallel 
traffic in either the out-the-window scene or the simulated Forward Looking Infrared (FLIR) sensor image. Note 
that traffic symbology rendered from ADS-B data was not considered equivalent visual contact. After EPs made 


the equivalent visual contact call, ATC cleared the EPs to land if the parallel traffic was still in sight. If the EP 
still had equivalent visual contact with the parallel traffic, the EP would respond verbally by saying “landing”. 
If equivalent visual contact was lost, EPs were instructed to call “go-around.” All data trials ended at the DA. 

3.1.2 Experiment conditions 

The experiment consisted of three display conditions. 

1. Condition A: A variable field-of-regard FLIR presentation (i.e. , the FLIR field-of-regard could be panned 
by the pilot in 15° increments, azimuth-only) shown on a head-down display. 

2. Condition B: A HWD concept which shows a conformal, full field-or-regard FLIR presentation with con- 
formal display of reported traffic, including the FIM-paired traffic. 

3. Condition C: A day-time visual approach without S/EV displayed. 

A total of 7 data trials were conducted per EP. Each display condition (A through C) was run twice for each 
EP for a total of 6 nominal runs. The seventh and final data trial was a staged off-nominal event that was not 
announced; however, during the briefing, it was stressed that a missed approach must be executed if equivalent 
visual contact with the parallel traffic was lost at or beyond the DA. 

The off-nominal scenario represents a case in which the parallel aircraft, while visible during the majority of 
the data trial, was not visible in the FLIR or with natural vision at the DA. The KSFO SOIA requires that the 
pilots flying the approach visually acquire the parallel traffic approaching 28L and the runway environment no 
lower than the DA. For the off-nominal condition, the proper response by the EP at the DA was to initiate a 
go-around and contact ATC. 

3.2 Experiment protocol 

All data reported is subjective. Numerous questionnaire data were completed by the EPs in a semi-structured 
process, both post-run and post-test. The questionnaires were designed to identify traffic awareness, workload, 
and approach situational awareness (SA). 

Post-run, EPs were asked to rate the following statements on a Likert-type scale (l=low; 10=high). 

1. Please rate the amount of overall traffic situation awareness experienced during the trial. 

2. Please rate the traffic situation awareness for the parallel traffic during the trial. 

3. Please rate the traffic situation awareness for the in-trail traffic during the trial. 

4. Please rate the amount of mental workload experienced during the trial. 

5. Please rate the ease of making the decision whether to continue the approach at DARNE. 

6. Please rate the situation awareness of in-trail spacing during the trial. 

Post-run, a Situational awareness rating technique (SART) 13 was used to assess overall SA for the run. After 
each data trial, EPs were asked to mark on a scale labeled from low to high their assessment (for that particular 
data trial) of 3 constructs: 1) Demand on attentional resources; 2) Supply of attentional resources, and 3) 
Understanding. 

Post-test, paired comparison questionnaires were administered to assess the constructs of 1) traffic situation 
awareness, 2) mental workload and 3) the decision to land . 14 ’ 15 For each of these 3 constructs, EPs compared 
the 3 display conditions: 1) head-down FLIR, 2) HWD and 3) Visual. 



3.3 Displays 

A simulated FLIR image was presented to the EP either on a head-down FLIR display or on a HWD (Fig. 5). 
The FOV of the head-down FLIR display was the same as the HWD FOV. The pan control for the FLIR camera 
(for those conditions) was pilot-controllable via a button pad. The nominal FLIR simulated E VS- visibility which 
met or exceeded the visibility required for the approach (i.e. , greater than 4 miles). 

The HWD condition had the same symbology as the head down FLIR display. The symbology consisted 
of a velocity vector, counter pointers for airspeed and altitude as defined by MIL-STD-1787B, 16 and a target 
locator box. The counter pointers were screen-referenced and the velocity vector and target locator box were 
earth-referenced. The target locator box was driven by simulated ADS-B and showed the traffic call sign on top, 
the vertical trend via an arrow and vertical speed in feet per minute. 

The head-down display for all concepts consisted of a Primary Flight Display (PFD), Navigational Display 
(ND) with Cockpit Display of Traffic Information (CDTI), Vertical Situation Display (VSD) and an Engine 
Indicating and Crew Alerting System (EICAS) display (Fig. 6). The PFD was rendered on a 21-inch diagonal 
LCD with an aspect ratio of 4:3 and the ND was displayed on a 17-inch diagonal LCD with an aspect ratio of 
5:4. The Head-down FLIR was rendered on a 15-inch LCD with an aspect ratio of 4:3 with no minification. The 
PFD was enhanced with SV for all concepts except for the visual condition. For the data trials with the visual 
condition, the PFD was a typical attitude indicator (blue over brown) with no terrain information. The vertical 
FOV of the PFD was 40°and the minification factor of the SV-PFD was approximately 1.4. The ND range was 
not controllable. 

The PFD also had FIM symbology which represented the commanded spacing speed for the paired in-trail 
aircraft. The ND showed the paired in-trail traffic as a highlighted chevron. The same in-trail traffic was shown 
on the VSD. Positional inaccuracies were added to the simulated ADS-B information. 



3.4 Results 

3.4.1 Traffic and situation awareness 

Table 1 shows the means and standard deviations (in parentheses) for the subjective measures. 

Post-run, awareness of the in-trail and parallel traffic was assessed using Likert-type scales (l=low; 10=high). 
An Analysis of Variance (ANOVA) revealed a significant main effect for measures for in-trail traffic SA, E(2, 17) = 
3.65 , p < 0.05; and parallel traffic SA, F(2, 17) = 4.03 , p < 0.05. The HWD condition was rated to provide 
equivalent awareness to the visual condition and both were rated significantly better than the head-down display 
condition. 





Figure 6. The PFD (left) and the ND/EICAS (right) head-down displays. 


Post-run, awareness of the FIM spacing interval and distance was assessed using Likert-type scales (l=low; 
10=high). An ANOVA was also conducted and there were no significant differences found between the display 
conditions, p > 0.05. Overall, pilots rated the display concepts very high for SA of distance interval and 
conformance maintenance and adherence with time-based ATC FIM clearance. 

An ANOVA was conducted on the SART measure. Similar to the results found for the questionnaire measures 
for traffic SA, there was a significant main effect for the SART, F( 2, 17) = 4.22, p < 0.05. The HWD condition 
was not significantly different from the visual condition but both were significantly different from the head-down 
display condition. 

Post-test, the Situation Awareness Subjective Workload Dominance (SA-SWORD) technique 14 confirmed the 
SA observed post-run, F( 2,8) = 4.75 , p < 0.05. 

3.4.2 Mental workload 

Post-run, mental workload during the data run was self-assessed using a Likert-type scale (l=low; 10=high). An 
ANOVA showed a significant main effect for mental workload across display conditions, F( 2, 17) = 3.87, p < 0.05. 
EPs rated the workload using the HWD condition the same as the workload for the visual condition; but, the 
HWD condition was rated lower than the head-down display condition. 

Post-test, the Subjective Workload Dominance (SWORD) technique 15 confirmed the mental workload post- 
run assessments, F{ 2,8) = 5.01 , p < 0.05. 


Table 1. The means and standard deviations for the situation awareness measures. 

Display Condition 


Dependent Measure 

Visual 

HWD 

Head-down Display 

Traffic Situation Awareness, post-run 

8.5 (0.65) 

8.5 (0.55) 

7.0 (1.25) 

SART, post-run 

110 (12.0) 

108 (15.0) 

85 (9.5) 

SA-SWORD, post-test 

0.956 (0.011) 

0.987 (0.020) 

0.765 (0.041) 

Mental Workload Questionnaire, post-run 

4.5 (0.5) 

5.0 (0.90) 

7.0 (2.0) 

SWORD (reversed scored), post-test 

0.957 (0.12) 

0.910 (0.03) 

0.652 (0.17) 

Landing Decision, post-run 

8.5 (1.25) 

9.0 (0.25) 

9.0 (0.25) 

Landing Decision Paired Comparison, post-test 

0.875 (0.15) 

0.850 (0.10) 

0.850 (0.10) 




3.4.3 Decision to land, approach SA 

Post-run, the EPs were asked to rate the ease of making their decision to land using Likert-type scales (l=difficult; 
10=easy). An ANOVA did not reveal any significant differences in terms of decision-making to continue the ap- 
proach or go-around at the missed approach point (DA) between the three conditions for either the questionnaire 
or paired comparison measures, p > 0.05. 

3.5 Discussion of results 

EPs commented they felt comfortable with the equivalent visual concept generally, and it provided the necessary 
SA to make a landing decision, at least for this task. The HWD condition was a similar task to the visual 
condition in that it required EPs to look left to acquire the parallel traffic visually, owing largely to the unlimited 
field-of-regard and conformality of the HWD which emulates closely to a pilot’s natural vision. 

The results showed that EPs rated the HWD and the visual condition the same in terms of SA and mental 
workload. This particular task required looking almost 90° to the left in order to monitor the parallel traffic. 
EPs commented that using the HWD used the same scanning motion as with the visual condition since the head 
tracker simulated the panning of the FLIR camera. Some EPs commented that they were able to detect the 
traffic easier with the FLIR though this may be a result of this particular simulation environment setup. For this 
simulation, the thermal contrast of the traffic was high compared to the visual scene used in this study although 
realistic given the simulated weather and proximity of traffic. In real world conditions, the contrast could possibly 
be poor in the FLIR image compared to the actual visuals. Conversely, real-world weather conditions, even under 
VMC, can result in poor acquisition and maintenance of visual contact with traffic using natural vision (e.g., a 
reason for the development of CDTI-assisted visual separation concepts). Therefore, the characteristics of the 
weather are important considerations in evaluation of any potential use of enhanced vision systems to support 
BTV/EVO operations. 

A few EPs liked the head-down condition as it was close to the PFD and ND; however, the head-down 
condition was rated as the highest workload and lowest SA. The head-down FLIR presentation could easily be 
incorporated into the scan pattern without having to look far off to the left. As the head-down task was in 
IMC, they were already head-down, thus it made monitoring the parallel traffic easier. However, the head-down 
FLIR required the EP to make constant camera pan adjustments. Also, the scenario flown did not have any 
turbulence. Some EPs were concerned that with only pan control, certain pitch and roll combinations would 
cause the parallel traffic to go outside the vertical FOV, thus requiring the need for pitch control. It was suggested 
that the FLIR camera auto-track the traffic to reduce workload on the pilot. Additionally, part of the preference 
for the head-down condition was motivated due to the prototype nature of the HWD implementation. The HWD 
was not optimized for commercial operational use and garnered some skeptical and negative responses. 

All EPs detected the off-nominal event and requested a go-around when visual or equivalent visual contact 
was lost with the parallel traffic. In the case of the HWD and the head-down conditions, the FLIR image was 
slowly faded completely out over a 10-second period for the off-nominal event only. This simulated a situation 
wherein the parallel traffic enters into a heavy moisture cloud that can not be penetrated by the FLIR sensor. 
For the visual condition off-nominal event, the 28L parallel traffic slowed down and was being overtaken by own- 
ship, resulting in the traffic not being visible out the window. In all cases, EPs felt they could easily determine 
whether contact with the parallel traffic was obtained and could be maintained and that the airport environment 
was in view so they could decide whether to continue the approach passed the missed approach point or call for 
a go-around as prescribed in the SOIA procedure. 

4. EXPERIMENT TWO: SURFACE OPERATIONS 

4.1 Methodology 

Prior the start of this experiment, 6 pilots participated in informal taxi sessions to provide feedback regarding 
the presentation of the SV airport on a HWD for surface operations. Also, their opinions were solicited regarding 
the display of the cleared taxi route on the HWD. From these informal sessions, pilots indicated their desired 
brightness level for both the SV airport as well as the cleared route. 



4.1.1 Evaluation task 

For the surface operations experiment, EPs were tasked to taxi a simulated large commercial transport aircraft 
from the runway to the gate at Chicago O’Hare (KORD) airport. All taxi routes began with the aircraft stopped 
on the runway and the active route having been transmitted. This taxi route was displayed as text as well 
as graphically on the Electronic Moving Map (EMM) and HWD. The taxi route on the HWD was displayed 
conformally as an augmented reality enhancement of the taxiway centerline. In other words, the computer 
generated cleared route line was placed on top of the actual taxiway centerline. All of the taxi scenarios were 
conducted in the daytime under variable visibility/fog conditions. Besides own-ship, all other traffic aircraft were 
on the ground and stationary. 

4.1.2 Experiment conditions 

The experiment varied 3 factors (described in detail in subsequent sections): 

1. Beyond-RVR concept that renders virtual airport objects (such as runways, taxiways, centerlines, tower(s), 
buildings) obscured by the weather in the HWD. In other words, the HWD virtually displays airport items 
and traffic that cannot be seen naturally and thus, not obscuring naturally visible objects. 

2. RVR visibility value. 

3. Traffic rendering style. 

4.1.3 Beyond-RVR condition 

The Beyond-RVR concept included 3 variations: 

1. A pilot-controllable condition where the EP was allowed to control the amount (via a knob in the cockpit) 
of the SV information that was rendered. 

2. A fixed condition where the pilot was not allowed to control the amount of the SV airport rendered; the 
amount of the SV airport rendered was automatically set based on the reported RVR. 

3. A total SV environment condition where the Beyond-RVR was disabled; the SV airport information was 
displayed everywhere (even over visible objects) and was not pilot controllable. 

The first 2 items are new concepts and were compared to the previous concept (total SV environment, Item 3) 
that was tested in a previous experiment. 5 

4.1.4 Visibility condition 

The outside visibility was varied over 3 ranges: 1) 300 RVR, 2) 600 RVR and 3) 2400 RVR. 

4.1.5 Traffic rendering style 

Traffic was rendered on the HWD in 2 different styles: 1) unfilled diamond and 2) generic aircraft model. The 
unfilled diamond model was a simple representation of an aircraft (using ADS-B symbol set) which showed 
aircraft position as reported by ADS-B. The second rendering of traffic used a generic aircraft model to depict 
other aircraft position and heading. This rendering is thought to be more intuitive but at the expense of increased 
clutter. 

4.2 Experiment design and procedure 

The experiment matrix was a full factorial design of 18 runs: 3 Beyond-RVR display concepts x 3 Visibility 
RVR x 2 Traffic Rendering. Each run averaged 3 minutes for 1.5 hours of data collection. 

The data runs were blocked into 3 groups of 6 runs. For each 6- run block, the visibility condition (300 RVR, 
600 RVR and 2400 RVR) was held constant while the Beyond-RVR and traffic rendering condition was varied. 
Following each 6-run block, a clutter questionnaire was given to the EPs. 

At the end of the 18 data runs, EPs were administered 2 SA-SWORD questionnaires. The first SA-SWORD 
questionnaire asked EPs to compare the 3 Beyond-RVR concepts for each visibility condition. The final SA- 
SWORD asked pilots to compare each of the 3 Beyond-RVR concepts regardless of visibility. 



4.3 Displays 

Figure 7 (left image) shows only the symbology on the HWD. As the HWD is see-through, black regions in the 
figure would be areas where “real-world” airport objects would be seen. The concept illustrated in Figure 2 
shows a better representation of the EPs view in the HWD. However, unlike the color SV imagery shown in 
Figure 2, the HWD used in the experiment was monochrome green. The symbology was developed from a 
previous experiment and has not yet been significantly altered based on the results of that experiment. The 
green ribbon depicts the cleared path and is not affected by the Beyond-RVR concept. It conformally overlays 
the ‘real’ yellow painted taxi centerline which allows the pilot to easily follow the route head-up. The SV airport, 
shown as the KORD tower, concourse buildings and airport surface in Figure 7, is also conformally displayed. All 
other symbology is screen-referenced. The brightness of the path was set to 80% opaqueness (0% is completely 
see-through). The SV KORD airport brightness was set to 50% opaqueness. The text “SR700” indicates that 
the pilot has selected the SV information to be clipped up to a 700 foot near clipping plane from the pilot eye 
point, thus the black portion at the bottom of Figure 7 would allow an unobscured view of the real airport 
environment. Other symbology shown is the ground speed (“17”), the current heading (“H350”), the current 
taxiway is boxed (“ F ”), the next route point (left turn, as denoted by the arrow, to “CNCS H”) and the path 
error (i.e., the localizer-type deviation scale and status symbology). The path error scale is detailed on page 42 
in the paper by Arthur, et al. 5 The filled green circle in the bottom left of the display is an indication to the EP 
that the head tracker is operating nominally. 

A head-down EMM display was rendered as shown in Figure 7. This display shows the airport surface from 
an overhead perspective. The display shows the cleared route, both graphically and textually, taxiway labels 
and airport environment. The range of the EMM was fixed to a range scale of 0.5 nmi. 
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Figure 7. The HWD (left) and the EMM (right) displays. As the HWD is translucent, black regions in the HWD figure 
represent see-through areas. 


4.4 Results 

4.4.1 Situation awareness 

Post-test, a SA-SWORD questionnaire was administered that compared the Beyond-RVR conditions with each 
of the 3 visibility conditions. An ANOVA on the paired comparison for display condition was not significant 
across either the three displays, or visibility, or interaction of display condition and the visibility, p > 0.05. 
Table 2 presents the paired comparison means and standard deviations. 

At the end of all data trials, EPs were asked which traffic rendering style they preferred. Eight out of 9 EPs 
preferred the generic aircraft model over the open diamond symbology. 




Table 2. The paired comparison means and standard deviations. 

Display Condition 


Visibility 

Pilot-Controlled 

Fixed 

Total SV Environment 

300 RVR 

0.545 (0.15) 

0.550 (0.10) 

0.550 (0.10) 

600 RVR 

0.450 (0.20) 

0.500 (0.50) 

0.550 (0.03) 

2400 RVR 

0.265 (0.22) 

0.301 (0.35) 

0.285 (0.43) 


4.5 Discussion of results 

All EPs commented the SV airport (i.e. , unlimited field-of-regard S/EV information), regardless of the Beyond- 
RVR concept tested, increased their SA. This finding is consistent with the results from previous experiments 
conducted with HWD for surface operations. 5 It was hypothesized that either the pilot-controlled or the fixed 
Beyond-RVR concept would be preferred to the total SV environment concept, but the results showed there 
were no differences in any of the Beyond-RVR concepts. Since this contradicts comments made from a previous 
experiment, those EPs who preferred the total SV environment were asked their opinion on the concept. They 
stated they liked having as much information as possible and at the low visibility conditions, there wasn’t much 
difference between the total SV environment versus a culled Beyond-RVR scene. To compound this further, the 
KORD airport model used to generate the out-tlie- window was identical to the perspective S/EV airport view in 
the HWD. In previous experiments, the airport models rendered in the out-the-window and the HWD differed. 
Since the airport models were the same in this experiment, there was no obscuration of “real-world” elements by 
design. If the out-the-window scene contained such elements as taxiway signs which were not part of the S/EV 
HWD view, it is hypothesized that the Beyond-RVR concepts would be preferred to a total SV environment. 

Regarding traffic rendering, EPs preferred the generic aircraft models rather than unfilled diamonds. EPs 
commented that the airplane model was easy to interpret while the unfilled diamonds lacked heading information. 
Further, while EPs felt the generic aircraft model was sufficient, some EPs felt that at least 2 models should be 
used to denote small and large aircraft. 

During two data trials, data collection was ended due the EPs reporting simulation sickness symptoms. 1 ' 
Simulator Sickness is a form of motion sickness experienced by pilots in either fixed or motion simulators from 
large FOV motion. This can be exacerbated by the total system latency inherent in head tracked HWDs. 
Simulator Sickness is experienced more with helicopter simulators than fixed wing simulators. 18 This is believed 
to be caused by large FOV visuals close to the ground with lots of motion (Vection illusion) and the lack of motion 
cues with coupling visual scene. We conjecture that helicopter motion is similar in many regards to fixed-wing 
large transport surface operations simulations. During ground turns, pilots must use very large head-motions to 
clear the turns and then the aircraft makes very large and rapid angular movements, especially in over-steering 
situations, to make the turns. Surface operations simulator sickness has been reported in previous tests 5 while 
no simulator sickness symptoms were reported in these same simulators for fixed-wing flight simulations. 

Both EPs who experienced Simulator Sickness reported the cockpit temperature to be “very warm” and 
had a sense of disorientation. One of these EPs reported having simulator sickness symptoms from previous 
simulations. As a result of the EPs experiences, the temperature of the room was reduced, the training protocol 
was changed and the number of data trials was reduced. After these changes were made, simulator sickness 
symptoms were no longer reported by EPs. 


5. CONCLUSIONS 

A HWD concept, coupled with a head tracker, was used to evaluate full-field-of- regard S/EV concepts for airborne 
and surface operations. A fixed-based simulation experiment explored the viability of using a HWD for equivalent 
visual operations during closely spaced parallel operations. In addition, surface operations were conducted to 
refine symbology concepts. 

For the airborne task, pilots rated using a HWD in IMC the same as the visual approach in terms of 
situation awareness and mental workload. These results are further evidence of the significant potential of 



BTV technologies to enable equivalent visual operations. Prinzel et al. (2011) provided initial evidence on the 
use of these technologies for delegated in-trail separation for nominal terminal movement area operations. The 
present study expands this research in showing potential value of these technologies for enabling equivalent visual 
simultaneous offset instrument approaches. Several simplifying assumptions were used in this initial study and 
further investigation and development is needed. In particular, the effect of realistic ADS-B sensor models and 
the development of EVS sensor technologies to meet the minimum performance levels envisioned in this operation 
must be evaluated. Additional steps would be to continue to evaluate the BTV concepts for other operations 
that could benefit from reducing or eliminating the visibility requirements to conduct the operations. Such 
examples include leveraging new approvals to conduct simultaneous dependent staggered straight-in operations 
to runways separated as close as 1200 feet, Simplified Aircraft-Based Paired Approach Concept (SAPA), and 
true “visual” approaches in which parallel traffic, to runways separated by less than 2500 feet, fly abeam one 
another and must maintain “visual” (or “equivalent visual” ) delegated separation. Other future directions include 
delegated equivalent visual separation for departures, and continued research on development of BTV concepts 
for equivalent visual surface operations and integration of these concepts across phases of flight and operational 
needs. 

For the surface operations study, there were no reported differences between the 3 augmented reality concepts 
or influence of natural visibility or on situation awareness or pilot workload. The pilot comments supported the 
value of BTV to increase capacity and safety in the airport movement area and that innovative approaches for 
intuitive display of S/EVS have merit and research should continue. Pilots did comment that they preferred 
traffic be rendered in the HWD as generic aircraft models rather than iconic (rendered using an open diamond 
shape). 

The encumbrance of the HWD was significantly less that previous NASA studies; however improvement must 
continue toward reducing the system latency. These results were obtained using a ground simulation facility with 
no motion effects. Motion simulation should be conducted to tease out the influence of vestibular stimulation 
and of course, these results must be fully validated during actual operations, focusing on daylight readability 
(particularly for evaluating the utility and capabilities for color), disorientation and illusion issues (i.e., “simulator 
sickness”), and augmented reality requirements for symbology and S/EV imagery. 
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